[SYSTEMDS-3863] Add PowerTransformer built-in functions - #2499

Summary

This pull request adds PowerTransformer built-ins with separate fit and apply workflows. The implementation supports Yeo-Johnson and Box-Cox transformations, per-column parameter estimation, optional standardization, and reuse of fitted parameters on new data.

Yeo-Johnson is the default method and supports mixed-sign inputs. Box-Cox is available for strictly positive inputs.

API

[Y, lambdas, means, scales] = powerTransform(
    X,
    method="yeo-johnson",
    standardize=TRUE
)

Y = powerTransformApply(
    X,
    lambdas,
    means,
    scales,
    method="yeo-johnson"
)

powerTransform estimates one lambda per column and optionally standardizes the transformed columns. powerTransformApply reuses the fitted lambdas, means, and scales without re-estimation.

Implementation

• Add powerTransform.dml for fitting and transforming feature matrices.
• Add powerTransformApply.dml for applying fitted transformations.
• Support Yeo-Johnson and Box-Cox transformations.
• Keep Yeo-Johnson as the default method.
• Support optional zero-mean, unit-variance standardization.
• Estimate each lambda by minimizing the negative log-likelihood with Brent optimization.
• Treat [-2, 2] as an initial bracket and expand it when necessary, allowing the optimum to lie outside the initial interval.
• Validate method names, fitted-state dimensions, and Box-Cox input requirements.
• Register both functions as SystemDS built-ins.
• Add entries for both functions to the built-in reference documentation.

Validation

mvn -Dtest=BuiltinPowerTransformTest test

Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
BUILD SUCCESS

The tests cover:

• default Yeo-Johnson fitting with mixed-sign and constant columns;
• Box-Cox fitting without standardization;
• Yeo-Johnson and Box-Cox optima outside the initial [-2, 2] interval;
• rejection of non-positive Box-Cox input;
• fitted Yeo-Johnson and Box-Cox apply workflows; and
• standardization state reuse.

Numerical outputs are compared with independent R reference implementations.

Documentation

User-facing signatures, arguments, return values, input requirements, and examples are documented in docs/site/builtins-reference.md.

Experiments

The reproducible evaluation compares no scaling, standard scaling, robust scaling, Yeo-Johnson, and Box-Cox on nine public datasets. Each experiment uses the fixed seeds 13, 37, 73, 101, and 149.

Supervised tasks use 80/20 train/test splits; classification is stratified and Parkinsons Telemonitoring uses subject-level group splits. Each transformation is fitted on training data and applied to test data. Clustering uses the full dataset.

The table reports the mean primary metric over five runs. A dash indicates that Box-Cox is not applicable because the feature matrix contains non-positive values.

Power transformations rank first or tie for first on four of the nine datasets. Representative results include:

• California Housing: Yeo-Johnson reduces RMSE from 0.6486 with standard scaling to 0.6114.
• Breast Cancer Wisconsin: Yeo-Johnson achieves the highest Macro-F1 at 0.9618.
• Seeds: Yeo-Johnson and Box-Cox tie for the highest ARI at 0.7759.
• Wholesale Customers: Box-Cox achieves an ARI of 0.5330, compared with 0.1873 for standard scaling.

Reproducible artifacts

• Experiment repository
• Experiment report
• Raw results
• Combined summary
• Figures
• English leaderboard workbook
• Dataset sources and citations

Limitations

The downstream effect of preprocessing is dataset- and estimator-dependent. The evaluation uses fixed distance-based estimators, three datasets per task, and no hyperparameter search or statistical significance test. The results therefore demonstrate practical cases where power transformations help, rather than universal superiority over other preprocessing methods.

Related issue

SYSTEMDS-3863